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ABSTRACT 

The Program for Student Achievement (PSA) focused on 
improving urban middle-grades student achievement in participating districts 
in Texas and California. Districts were asked to establish performance 
targets for eighth graders in core academic areas. The PSA conducted a 
quantitative evaluation of districts ' progress and produced an annual report 
for each of the districts using data from National Assessment of Educational 
Progress-based assessments, longitudinal reports of state and local 
assessments, and longitudinal tracking of student progress. A 2001 seminar 
was held to review districts* experiences and lessons learned from working 
with the targets. Lessons learned included target setting requires a 
thorough, shared understanding of what "meeting standards" means; alignment 
of standards, assessments, and instruction is fundamental to performance 
targeting;- staff development on understanding and using assessments is 
important; and community engagement is crucial. Mechanisms for promoting 
personal and institutional accountability included promoting shared goals and 
focused reporting of student progress. Challenges to using performance 
targets included an absence of longitudinal data and continually changing 
policies. Alternative strategies for targeting improvement and change include 
giving teachers' perspectives greater weight in setting standards, providing 
diagnostic information that makes sense to teachers, and increased 
specificity in the way goals and plans are articulated. Included is an 
example of performance targeting that is working under the direction of the 
Southern Regional Education Board. List of participants and a meeting agenda 
are appended. (SM) 
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V 



An important feature of the Program for Student Achievement, as sponsored by the Edna 
McConnell Clark Foundation (EMCF), is the performance targets that participating school districts have 
established in key academic areas. The targets were designed to serve as goals for middle-grades student 
achievement, with the expectation that meeting the goals would require improved district performance in 
teaching, administration, professional development, parental involvement, and other areas. The 
Foundation expected that these targets would be ambitious but reasonable incentives for districts and 
schools and that they would stimulate significant systemic changes in school functioning and gains in 
student achievement. While participating school systems and schools accomplished some of what they 
sought during this period, it is evident that districts set unrealistically high performance targets and that 
these targets became ideals that could not be used practically as guides to improved implementation. 

Periodically during their collaboration with the Clark Foundation, districts and Foundation 
consultants re-examined goals and redefined milestones, targets, indicators, and measures for assessing 
progress toward the targets. Although the districts participated fully in these reflection and revision 
processes, the effort did not stimulate the level of commitment to achieving either the targets or the 
interim indicators that the Foundation had hoped. 

As the Foundation concludes its work with school reform, it is examining the lessons learned by 
its grantees and other colleagues, so that these lessons can be shared with the field. To review the 
districts’ experiences with the targets and recommend strategies for the future use of educational 
performance targets, Policy Studies Associates (PSA) convened a seminar on June 25, 2001, in 
Washington, D.C. Meeting participants included Foundation representatives, invited consultants, and 
representatives of the Corpus Christi Independent School District (CCISD), the Long Beach Unified 
School District (LBUSD), and the San Diego City Schools (SDCS). The participant list and meeting 
agenda are appended to this report. 

The agenda for the meeting asked participants to address the following issues and questions: 

• What did participating districts learn from the process of establishing performance targets 
and monitoring progress toward the targets over the past five years? 

• Was it appropriate to use standardized assessments as the basis of performance targeting? 
Did it help districts achieve improvements in student learning? 

• What were the conditions that limited the effectiveness of standardized assessment tools 
as the means of tracking district performance? How can those conditions be reversed or 
overcome in the future? 

• To what extent should local targets link to state targets for student achievement gains? 
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What alternative mechanisms would promote the personal and institutional accountability 
that were sought through the use of performance targets? 

If performance targeting using standardized tests holds promise, what additional 
measures of student achievement would be useful in promoting successful 
implementation of educational reform? 

What lessons from the districts’ and the Foundation’s experience with performance 
targeting should be shared with other education-reform initiatives? 



This discussion summary begins with a review of the background and context for performance 
targeting in the three Clark districts and summarizes the districts’ experiences implementing the targets 
they set. It then reports on a model of performance targeting that is working under the direction of the 
Southern Regional Education Board. The summary concludes with ideas that surfaced from the 
discussion of the dilemmas affecting educational performance targeting, especially in high-stakes 
accountability environments, and themes about accountability approaches that emerged during the day’s 
discussion. 



Background and Discussion Context 



Context for Performance Targeting in Clark Sites 



Hayes Mizell, Director of the Program for Student Achievement, opened the meeting by 
summarizing the history and expectations that would frame the day’s discussion. His comments provided 
historical perspective and focused the agenda. 



According to Mr. Mizell, six years ago, the Program of Student Achievement invited six urban 
school systems to submit proposals describing how they would use academic standards to leverage the 
reforms of all their middle schools and increase academic achievement. In the request for proposals, the 
Foundation asked districts to establish performance targets for eighth-graders in the core academic 
subjects of language arts, mathematics, science, and social studies. Underlying this request were a set of 
working assumptions that framed the Foundation’s vision and hope for the five-year reform effort: 

• Standards and assessments would be aligned with performance targets . The 

Foundation assumed that, within districts, there would be a consensus on standards linked 
to performance targets. “We assumed that it was possible for school systems to establish 
what it meant to perform at standard and to translate that concept into a quantitative 
target, using assessment methodologies of their choice,” said Mr. Mizell. The 
assessments that districts would use over time to determine students’ progress toward 
performing at standard would remain consistent from year to year. 
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• Performance targets would be aligned with curriculum , organization , and leadership 
reforms . The Foundation expected that student performance targets would motivate 
coordinated improvements in instruction, curriculum, leadership, and student support, 
enabling students to perform at standard and meet the performance targets. According to 
Mr. Mizell, the Foundation anticipated that commitment to these reforms would become 
“ the priority, not just a priority” that would drive district decision making in these areas. 

• Performance targets would serve as reform levers . Target setting was expected to 
stimulate a systemic focus on teaching and learning that would achieve desired academic 
outcomes. “We assumed the student performance targets, established at the beginning of 
the Program’s six-year initiative, would focus the school systems and their middle 
schools on the academic outcomes they were seeking to achieve, and that the school 
systems would in turn use these targets to mobilize and inspire faculties and citizens to 
achieve desired levels of student performance,” said Mr. Mizell. 

• With support from the Foundation and others , districts would have the capacity to 
carry out their reform visions. Districts would have the technical expertise and the 
capacity — “the will, the knowledge, and the skills” — they needed to both create 
conditions that enabled students to meet standards and also implement whatever 
initiatives they designed and adapted. 

Mr. Mizell indicated that, at the outset of the Foundation’s program activities, there were high 
expectations for district-level flexibility and autonomy. Program sponsors believed that, given the 
encouragement and resources, the districts would use student performance targets as a means to foster 
internal school-level accountability and advance improvement and reform. The six years have, however, 
taught the Foundation and its collaborators a new appreciation of the complexity of the issues involved 
with putting good intentions for improvement into action using performance targeting. 



Mr. Mizell noted that one way to strengthen student learning is for districts to demonstrate their 
commitment to high standards through meaningful goal setting and self-monitoring. In some states and 
districts, the availability and visibility of student performance data produced by these systems have 
contributed to what appears to be an increased national emphasis on instruction in literacy, among other 
areas of learning. Furthermore, these local successes, although spotty, have focused new attention on the 
achievement gap among various racial and ethnic groups. 



Based on Mr. Mizell’s experiences during the past five years, the accountability focus of the 
Foundation’s program may have been founded on questionable assumptions. He expressed hope that as 
the national commitment to accountability-driven systems strengthens, policy makers will recognize that 

internal accountability is just as important, if not more so, than external accountability. 

... the national and state governments are defining and measuring what they believe represents 
adequate student learning and they are sharing the results with the public at large. But what about 
the school systems and the schools? What do they believe? [What about] their internal 
accountability? What are the acceptable levels of student performance toward which school 
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systems want to strive? What are school systems' student performance goals, regardless of those 
established by the state? 

Hindsight suggests that performance targets did not have the reforming or motivating impact that 
had been anticipated in the Clark-sponsored initiative. Experience reasserted the obvious: that 
performance targets are only meaningful if students are engaged in learning of high-quality subject 
content and benefit from high-quality teaching. Mr. Mizell added, “When the performance goal becomes 
divorced from these two critical opportunities to learn, it becomes just one more distraction that impedes 
rather than benefits students’ academic development.” 

The purpose of the day’s conversation was to reflect on the lessons learned from the Foundation’s 
performance focus and to learn from its experience, especially in light of the continuing national 
commitment to performance-oriented, accountability-based reform. Acknowledging that “sometimes 
clarity comes after hard experience,” Mr. Mizell saw this meeting as an opportunity to gain insight and 
new information from the discussion, if not to arrive at any ultimate answers. 



Context for the Evaluation of the Program for Student Achievement 

Elizabeth Reisner summarized the longitudinal evaluation that the Foundation conducted in 
participating districts through PSA and Education Matters. Education Matters conducted an on-going 
qualitative analysis of the standards-based reforms. Each year, Education Matters visited a sample of 
schools and classrooms and interviewed the districts’ curriculum and professional development 
strategists. Based on these data, Education Matters annually reported the status and challenges of reform 
implementation in each district to the districts and to the Foundation. 



PSA conducted a complementary quantitative evaluation of districts’ progress and produced an 
annual report for each of the districts and the Foundation. PSA used three types of information in these 
reports: 



NAEP-based assessments. Beginning in 1998, PSA developed standardized reading and 
mathematics assessments based on public-release items from the 1994 National 
Assessment of Educational Progress (NAEP). In cooperation with participating districts, 
PSA administered these assessments over four years to randomly selected samples of 
eighth- graders in each of the Clark reform districts. Each year, PSA analyzed and 
reported student performance within each district on the NAEP-based assessment. 

Longitudinal reports of state and local assessments. PSA worked with each of the 
districts to obtain, analyze, and report state and local achievement data. In the course of 
this work, PSA evaluated the data in conjunction with the NAEP-based scores. 
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• Longitudinal tracking of data on student progress . PSA also worked with the districts 
to strengthen their capacity to track students’ academic progress longitudinally, as they 
move from middle school through high school, so district planners could reflect back 
on — and improve — program quality in sending middle schools. 

Ms. Reisner pointed out that longitudinal tracking of student progress was a challenge for 
districts. Although the Clark districts were in varied stages of developing their individual student- 
tracking systems when the PSA evaluation began, the systems did not support reporting and analyses that 
would track the progress of individual middle-grades students as they moved through high school. 
Furthermore, districts’ top managers did not regard the development of longitudinal analyses of school 
progress as a priority. From the Foundation’s point of view, reporting evidence of student progress from 
middle schools through high school significantly complements other outcome information. To encourage 
this reporting, the Foundation offered districts small supplementary grants and technical assistance from 
PSA to strengthen their capacity for longitudinal reporting. Although three districts successfully carried 
out longitudinal reporting activities with this supplementary funding, the projects did not alter their long- 
term outcome-reporting practices. 



What Did Districts Learn from Performance Targeting? 

The representatives of the three Clark districts described their experiences with performance 
targeting over the past five years. Participants incorporated the agenda questions into their discussion of 
what they learned in establishing targets, what other mechanisms they found to promote accountability, 
what continuing challenges they faced, and how they might improve the process. 



Issues in Establishing Performance Targets 

In separate presentations, the three districts described what they learned from their experiences 
with performance targeting, and identified supportive activities, barriers, and strategies they used to 
address the barriers. Similarities across the districts include those described below. 



Understanding and teaching standards. District teams observed that target setting requires a 
thorough and shared understanding of what “meeting standards” means. This concept is not clear for 
teachers, school leaders, or for the public, district leaders emphasized. Long Beach’s Lynn Winters 
pointed out that standards are typically vague statements of content and skills that students are expected to 
master and “their existence does not change the nature of teaching and learning.” For standards to be 
useful, teachers must see the linkage between standards and curriculum. Attaining this knowledge 
requires teachers and administrators to share an understanding of the instructional processes needed to 



5 



Proceedings: October 2001 



7 



help students attain them, but such understanding is difficult to “backwards engineer.” Under the best of 
circumstances, districts align their diagnostic tools with standards and assessment targets so both teachers 
and students recognize the gaps between what students need to know and what they actually know. 
Making these linkages, however, is far easier to conceptualize among a few standards-setters or 
assessment developers than they are to make happen in an environment driven by many stakeholders. Ms. 
Winters observed: 

Just that you have standards and 200 people buying into them and signing off on them. . . doesn’t 
mean anything’s going to change. The standards are really operationalized by the assessments, 
assignments, and other student progress indicators. That work takes years and the thing we 
struggle with is [attaining the] shared view of the teaching and learning process. 

Aligning standards, assessment , and instruction. The alignment of standards, assessments, and 
instruction is fundamental to performance targeting. Discussants pointed out that the potential benefit of 
performance targeting was complicated by the challenges of developing and implementing each 
component and aligning these complex and interdependent elements. There are many performance 
indicators in a standards-based system — grades, report cards, state and local assessments — each providing 
different measures of achievement. Karen Bachofer observed: “We should never use a single measure to 
make high-stakes decisions for students. . .. We want to use multiple measures, but we’re finding that’s 
tricky because different measures often say different things.” Even formal assessments can provide 
conflicting information. For example, some reading assessments provide information about students’ 
decoding skills, reading fluency, and literal comprehension (“the basics”), while others focus on students’ 
ability to make inferences, analyze and interpret text, or synthesize information from multiple texts. 

Test quality is also a stumbling block. Ms. Winters pointed out that because a test is state- 
sponsored does not ensure it is of high-quality, valid, or instructionally useful. For example, California’s 
norm-referenced state assessment is not suitable for linking with day-to-day instruction. “So when you 
talk about linking our programs with the state assessment program, there’s got to be a fair amount of 
caution,” Ms. Winters added. 

A further challenge, the San Diego team observed, is that many teachers have instructional 
expertise but they do not have experience interpreting and applying standardized assessment information 
to classroom instruction. Often when teachers receive assessment information, they look at the product— 
the scores — rather than the knowledge that students have or have not mastered, focusing on the “targets in 
the aggregate, not the disaggregate,” explained Ms. Bachofer. She has found that many teachers have 
difficulty deciding whether to concentrate on “the forest or the trees. Can a student decode a piece of 
text? Do they really get the gist of it, do they understand it, can they use it?” To complicate matters, 
when the pressure is on, there is a temptation to reduce ideas “to the lowest common denominator and the 
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effect is to fragment instruction into little bits and pieces of information about student achievement rather 
than to keep focusing on the larger picture.” 

Corpus Christi found that the district leaders can draw the linkage between standards and 
assessments, but many teachers persist in using the traditional in-class strategies that they have always 
used. Katherine Conoly reported an experience familiar to staff developers: 

... we would take the standards, we would put them out there, and we would do considerable staff 
development. But then teachers would go back to that filing cabinet and find those old 
assessments that they’ve always used, they liked using, and were comfortable with using. And it 
didn’t matter that it they didn’t match up with standards! 

CCISD staff responded by distributing school and classroom-based reports, Look Backward Look 
Forward \ to update principals and teachers on how currently enrolled students performed on the previous 
year’s tests. The reports were distributed for the first time in summer 2000 and were provided again in 
August 2001. In Conoly’s view, this historical information, organized by current classrooms, pushes 
accountability to the teacher level, showing principals and classroom teachers how accountability 
reporting can actively inform instructional decision-making. 

Contradictions between state and local targeting systems. In both Long Beach and San Diego, 
schools are struggling to understand how state-established targets (the Academic Performance Index, or 
API) are connected to what individual students actually know and are learning. Target setting and 
monitoring must start with individual students and “work up” — not the other way around. Yet John 
Novak explained how the California accountability formulas push locally designed instructional reports to 
take a back seat to the priority of meeting the API target. In response, the Long Beach research office 
began to create student-focused data, but their schools wanted to know what changes would improve their 
API numbers. As a result, the district is now providing schools with reports that show how many students 
must move up from one quintile to the next and in what demographic categories. This way, “they can 
improve their API numbers,” said Mr. Novak, but that “may or may not be related to improving 
achievement. I don’t think it’s very effective information. In short, the targets in the California system 
cause schools to focus on district and aggregate numbers of students who are English proficient, rather 
than on what and how to teach students who are not proficient.” 

Ms. Bachofer emphasized that teachers’ “ownership” of performance targets is necessary 
because, unless teachers see them “as believable, as valid, as comparable, and as accessible, they don’t 
own them and they aren’t likely to change their practice.” Ms. Winters added that California prohibition 
against using practice materials matched to the SAT-9 discouraged teachers from practicing tested skills 
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with their students, “so it’s really not clear how explicit the link between district and state standards really 
should be.” 

In Long Beach, one solution “was to take our district assessments and [statistically] correlate 
them to the state assessments. We told the teachers, ‘You're not allowed to teach the SAT-9 but you are 
allowed to teach the Long Beach local tests.’” District leaders tried to persuade teachers that if they used 
the locally developed materials and kept their instruction focused, the payoff would be positive state 
assessment results. But this is “a hard message to sell,” Long Beach’s Geno Flores pointed out, “because 
the assessment we're giving them didn’t look like what the state had given them.” 

Staff development on understanding and using assessments. Since the Clark collaboration 
began, districts have put coordinated assignments and assessments in place, but none of the districts was 
as successful as it wanted to be in reaching teachers and schools with systematic staff development about 
these resources. Corpus Christi concentrated on developing teachers’ understanding that standard s-based 
teaching is substantive, linked to life experiences, and focused on specific content targets. Corpus Christi 
also found, in working with the Foundation’s consultant, George Perry, that professional development 
must be personalized. Giving the same treatment to everyone makes no sense when they are on different 
points of a learning curve. 

Long Beach focused its performance targets not on student test results but on building teachers’ 
understanding of goals and standards and on creating assignments and assessments to reflect them. They 
also sought to strengthen teachers’ capacity to design challenging learning activities and common criteria 
forjudging the quality of students’ responses, as well as to evaluate the quality of teaching and to plan the 
next steps for students. 

Each of the districts made the same kinds of erroneous assumptions about what educators know 
that Mr. Mizell described in his opening remarks, and these assumptions led to some false starts. Ms. 
Conoly acknowledged that the district leadership failed to recognize early enough that principals and 
teachers had limited facility in working with data, so the staff development should have focused on two 
elements, she said: (1) helping teachers create assessments and strengthening principals’ capacity to 
recognize and encourage high-quality assessments; and (2) teaching principals and teacher leaders to 
apply data from assessment and other evaluation activities in instructional contexts. 

Community engagement. Districts need public support to implement their reforms, and 
community engagement demands an on-going dialogue about student progress toward explicit and 
understandable targets and goals. But San Diego learned that “when targets get personal, people get 
cranky,” reported Ms. Bachofer. “A lot of people are finding out that their students — the ones they 
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thought were above average — really aren’t.” Parents do not understand the standards, especially if there 
is a discrepancy between their child’s report card grades and their assessment scores, and they object 
when standards become an obstacle to promotion. Ms. Bachofer cited two examples of how San Diego is 
engaging the public and developing relationships to support change. First, schools have asked parent 
coordinators to serve as liaisons between the community and the school; second, schools are drawing up 
learning contracts for at-risk students, which the students, parents, and the principal and teacher sign to 
confirm agreed-on expectations. 

Despite efforts at outreach, many members of the public regard performance targeting as a top- 
down effort that excludes them from the process. In San Diego, the reform effort is highly visible and 
one public interest group or another objects to almost every initiative, according to local representatives. 
The district, after several years, has aligned the system — school-level instructional leaders, principals, and 
professional developers are now working from the same framework — but the public is not convinced of 
that coherence. 



Mechanisms for Promoting Personal and Institutional Accountability 

The participating districts suggested alternative mechanisms to promote the level of personal and 
institutional accountability that are expected of performance targets. 

Promoting shared goals . Structures must be in place to ensure the consistency of and support for 
accountability. San Diego found that 243 different reading programs were used K-12 when the reform 
began. “It’s no wonder the kids were confused,” reported Ms. Bachofer. Through the Blueprint for 
Student Success, the administration narrowed the program focus to make sure that, throughout the system, 
programs are aligned and moving in the same direction. 

Corpus Christi’s Maria Goodloe recommended the following steps to generate shared 
commitment and set targets: incorporate greater definition and depth to the performance targets, specify 
data sources in the beginning for each target, limit the number and scope of targets, build shared 
accountability, and create crosswalks to the state accountability system to avoid any disconnect. In 
addition, Corpus Christi encouraged collegial collaboration and shared responsibility for middle-grades 
instruction and achievement to replace campus competition. 

Districts also took additional steps to achieve alignment and engagement, so that performance 
targeting was understood outside of the central office. They included the following: 
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• Aligning middle school targets and the state system. Long Beach worked to integrate 
the state system and their middle school targets, establishing hard “bottom line” targets 
for students, such as: an eighth-grade student will not advance to high school if he/she 
receives two or more F grades. This target was much more concrete than their original 
achievement target (75 percent of eighth-grade students will meet standards), but was put 
into place after the 75 percent target was set. In retrospect, a revision of their Clark goal 
to focus on the meaning of middle school grades could have made their targets clearer to 
teachers, Ms. Winters reflected. The focus on the “two F policy” could have motivated 
teachers to embrace common assessments and assignments for students or at least 
common criteria. 

• Consistent guidelines . According to Sally Bennett, San Diego lacked consistent goals 
across their demographically diverse district. The district worked to develop a consistent 
set of guidelines for schools on (1) making decisions about student placement, promotion, 
and retention and (2) making sure that youth have access to the courses and the support 
systems that they need. San Diego produced a notebook that is based on the district’s 
Blueprint for Student Success. Principals received this handbook, and it is also posted on 
the district’s website. 



Focused reporting of student progress. Report cards in each of the districts are being revised to 
better inform parents, students, and teachers about standards-based progress. Corpus Christi is providing 
parents with information about students’ level of performance in content areas and their progress toward 
mandated performance standards. Long Beach has developed a student academic profile to report 
assessment and report card information that is maintained in the district’s database. Included in these 
student profiles are: students’ ELL status, their attendance history, information on their district 
assessments, writing history, math background, math standards assessments portfolio, mathematics end- 
of-course exam, open-ended math assessments, course history, interventions, and reading progress. Mr. 
Novak reported that his staff are creating procedures for recording course taking progress, as well. 
Teachers receive these reports first, and they distribute them at conferences with parents. San Diego is 
also developing a standards-based reporting system, which is expected to be implemented in the next 
year. 



Increased coordination across offices , departments , and schools for implementing standards- 
based instruction . Corpus Christi leaders observed that no single central office department should by 
itself be responsible for accountability. The districts’ research offices have worked increasingly closely 
with the district instructional directors to insist that schools do more than give students a pep talk, make 
sure they have breakfast, and “then sit back and hope for the best.” They are coordinating their 
professional development so that schools learn their students’ educational needs by working with the data 
and designing local assessments to keep informed about students’ continuous progress. 

Districts gave examples of other mechanisms that may promote personal and institutional 
accountability for student achievement, including principal evaluation, campus action plans, external 
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evaluation, dissemination of best practices, cross-school visits, and principal and teacher presentation at 
local, state, and national levels. 



Challenges to Using Performance Targets 

Retrofitting ongoing work into a new framework is an unsatisfactory way to try to reduce the 
complexity of this work. Ms. Winters said that LBUSD should have been more realistic about the time it 
takes to engage teachers and leaders in devising and carrying out plans. In retrospect, their performance 
targets were a strategic plan, not a performance targeting document. It would have been clearer if all 
targets had been stated in terms of student achievement, with subsidiary goals for staff development, 
resources, and infrastructure. Instead, LBUSD’ s target document had one global student achievement 
goal and five process or input targets. 

Absence of longitudinal data systems . Each district said that their current data systems do not 
support longitudinal analyses or shared work. For example, in San Diego, principals and teachers do not 
have access to on-line/continuous progress achievement data on students to inform instructional decision- 
making. It has taken five years to bring Long Beach’s student profile on-line and, after two years, Corpus 
Christi school sites are just beginning to understand the benefits of longitudinal reporting of TAAS 
results. Despite the supplementary assistance the Foundation offered districts, none of the three has “real 
time access” to integrated, longitudinal information on student progress. 

Continually changing policies . Forces associated with the state accountability system, 
professional development initiatives, and legislation also shape or challenge district work. Shifting 
assessment/accountability requirements can make target setting difficult. Initially, San Diego used a 
portfolio system as part of its assessment program, but the district had to change its system when the state 
changed the state assessment. As a result, it is now developing assessments as part of a locally controlled 
assessment system that will not shift with state-level changes. Alignment with state performance targets 
is only part of the story, however. Many state assessments are not reliable for certain purposes. For 
example, the state mandates annual administration of the California English Learner English 
Development Test (CELEDT). The information the CELEDT provides cannot drive instructional 
decision-making, however, and is costly. 

Trade-off between resources and time for professional development The need for additional 
resources and professional-development time was common across districts. Long Beach described the 
tension that they experience from engaging teachers in assessment development and taking teachers away 
from instructional time and other staff development opportunities. Mr. Flores noted that assessment 
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development is time-consuming work, which requires much input from teachers and field-testing of many 
items. Ensuring teacher participation is a juggling act, requiring that teachers be brought together during 
release time, the summer break, or inter-session time. Both San Diego and Long Beach faced even 
greater tension when the state took away their staff development days while also imposing an 
accountability system with serious consequences for low-performing schools. 



Alternative Strategies for Targeting Improvement and Change 



PSA invited three assessment and accountability consultants to review the work of the 
participating Clark districts and to suggest alternative mechanisms for promoting district- and school- 
level accountability. Each of the consultants was assigned a district whose materials they reviewed. The 
consultants were: Lloyd Bond, Professor, Educational Research Methodology Department, University of 
North Carolina; Michael Feuer, Director, Center for Education, National Academy of Sciences; Suzanne 
Triplett, Director of Operations for NAEP, National Center for Educational Statistics; and Gene Bottoms, 
Director, High Schools That Work, Southern Regional Education Board. Mr. Bond, Mr. Feuer, and Ms. 
Triplett were paired with Long Beach, San Diego, and Corpus Christi, respectively. Mr. Bottoms made a 
presentation on High Schools That Work and provided general comments on all the districts’ activities. 
Based on district presentations, the consultants identified alternative strategies for targeting improvement 
and change. These strategies are as follows: 

• Defer to teachers , not to tests . According to Mr. Bond, the public defers too willingly to 
psychometric experts, “What we typically do, we tend to throw a veil of numbers over 
what actually goes on.” A preferred strategy would give teachers’ perspectives greater 
weight in standards setting. In Mr. Bond’s view, teachers are best able to determine the 
information that they need, such as what their students should know in order to move on 
to the next grade. 

• Recognize that setting standards and raising student achievement are not equal 
Setting performance standards is just one tool for raising student achievement. Mr. Feuer 
said that the investment in test-based evaluation may be misplaced, and people are 
putting too much faith in what the external measurements say. 

• Fix the system, not the blame. According to Mr. Bottoms, performance targets diagnose 
students' needs but they do not explain “the system that produced the youth” with these 
needs. He is interested in analyzing “what’s wrong with the system, what is working 
with the system, what is not working with the system.” In addition, finding out about 
student experiences and classroom behavior may yield insight into what raises student 
achievement — for example, by showing that doing homework makes a difference. 

• Provide diagnostic information that makes sense to teachers . The experts agreed with 
the district leaders that teachers need more diagnostic information about students. 
According to Messrs. Bond and Feurer, however, it is hard to draw diagnostic 
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information from multiple-choice exams. Mr. Bottoms has seen the power of providing 
teachers with information about what students need to know to succeed in the next grade. 
Mr. Bond distinguished between teaching to the test and teaching from the test. Teaching 
from the test involves having the actual items from tests available for instructional 
purposes. But this is almost impossible, given that tests keep these questions 
confidential. Teaching to the test involves familiarizing students with the structure of the 
test and the questions, and teaching the content domains that the test measures. 

• Understand that multiple measures of learning can be useful . Mr. Feurer suggested 
that multiple measures could support standards-based learning because different 
measures provide different analyses of similar academic challenges. The challenge is in 
teaching people how to use these analytical tools to the full extent. 

• Increase the specificity of goals , expectations , and focus . Ms. Triplett said that wording 
in accountability plans sometimes obscures who is responsible for what. She suggested 
Corpus Christi’s “wording is too soft.” She encouraged greater specificity in the way 
plans are articulated. More often than not, the message is that what can be measured is 
the only thing that counts. Ms. Triplett exhorted districts to raise the bar on expectations, 
to clarify what is meant by success and examine mastery of grade level performance — 
going beyond minimum expectations. Mr. Bottoms expanded on this idea and said 
districts should look beyond school and attend to what they are preparing students for. 



The SREB Model: Performance Targets That Work 

Features of the High Schools That Work (HSTW) Program 



Mr. Bottoms has spent the last ten years working with high schools. Two years ago, he and his 
colleagues received funding from the Clark Foundation to apply what they learned in high schools to 
middle schools. As they did in HSTW, the SREB team is using NAEP-based assessment results and 
student and teacher survey data to focus instructional strategies and assessments more closely on gauging 
student progress. 

Mr. Bottoms has found that, by setting both student performance and school- and classroom- 
practice goals, both the school system and students are considered as part of the change process. 
Changing students’ experiences in schools offers one way to increase achievement. “Believing that ... if 
you wanted to change what students look like, you must change the experiences they have at high 
school,” Mr. Bottoms said. To this end, HSTW examined students in the career/vocational sequence, 
those “not taking the so-called advantaged curriculum,” and concentrated its efforts on eliminating the 
general track in high schools. 
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The HSTW team analyzed assessment systems and found that most fail to (1) provide information 
about which school and classroom practices are working, (2) go beyond collecting demographic data to 
collect data on student experiences in schools, and (3) connect students’ school experiences with student 
performance. In response, they created a new assessment system with the following features: 



• Ambitious and focused reading , mathematics , and science performance goals linked 
with survey data about opportunities to learn . The program set performance goals high 
enough that, when students meet them, they avoid remedial studies in college and are 
assured of passing employer-administered exams. In addition, when the NAEP-based 
reading, math, and science exams are administered, students complete a 200-item survey. 
“We think eighth-grade students are pretty accurate customers. We want them to tell us 
about their experience in schools.” The faculty are also surveyed, and their perceptions 
are reported back to schools as well. 

• School-practice goals . The program defined school goals as that of changing conditions 
so that students would be able to perform at standard. Using the results of student and 
faculty surveys, schools receive reports that connect what occurs in classrooms with 
student performance on NAEP-based assessments. The report connects performance to 
data on what courses students take, what is expected of students, how students are taught, 
what types of extra help students receive, and benchmarks to compare each school 
against all similar schools in the network. “Faculty began to take on these data and began 
to take the information seriously,” commented Mr. Bottoms. 

• Providing “cover” to principals who have the courage to change the system “It’s a 
lonely job out there,” Mr. Bottoms finds, “so school principals need a lot of coverage to 
do what’s right.” HSTW protects principals by collecting and providing the information 
their teachers need but the districts fail to provide. With the results of the survey data 
they collect, the HSTW team sit down with faculty and point out that students who 
perform well on tests say they read a lot of books, write a paper in every class every 
week, and have their language arts performance displayed throughout the building. 
Principals report that the data are both substantive and compelling, so that teachers begin 
to talk about student achievement in the context of specific ways they can alter their 
instructional practices. 

• Training in analyzing data reports for teams of principals and teachers . The program 
builds internal accountability by training principals and teachers in teams. Training in 
teams ensures a common understanding of how the system is failing and what to do about 
it. The teams receive research briefs indicating what the research says about how to 
connect different instructional practices with student achievement. 

• Links to policy . State policies matter. “It’s awful difficult to reform schools without a 
policy framework,” cautioned Mr. Bottoms. HSTW convenes policy makers annually to 
look at the lessons learned from the data and to link those lessons to policy. Policy 
changes have helped principals convince teachers of the need for change. For example, a 
North Carolina principal reported that, without the high-stakes end-of-course exam in 
Algebra I, he could not convince his best math teacher to teach Algebra I. 
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Multiple indicators to raise raising expectations . HSTW indicators measure the extent 
of faculty perceptions and extent of alignment of the vision and mission of what the 
school is about. “Our mission is to teach all kids what we used to teach to all the best 
kids, and we try to prepare both for further study and employment.” Participating schools 
continually receive indicator data to maintain their awareness of student needs. 

Indicators include the courses students take in high school, the practices teachers use in 
their classrooms, and information about principals’ leadership practices. In addition, the 
program keeps schools informed about strategies for engaging the public and for ensuring 
that students and parents are aware of the school’s goals and expectations. 



Implications for Middle-Grades Reform 



The HSTW program has given Mr. Bottoms and his team a unique perspective on the predictive 
elements that enable middle-grades students to succeed in high school. Mr. Bottoms noted that middle 
schools face an uphill battle because they receive fewer resources than either high schools or elementary 
schools. Funding, however, is only part of the issue. In many states, middle-school teachers are not 
certified in the subjects they teach. On the state level, there is typically no full-time person in the state 
education department responsible for middle-grades issues. The public debate has been about readiness 
for first grade and college but not for high school, although this too is a difficult school transition. The 
HSTW model of performance targeting offers approaches that have implications for middle-grades 
reform: 



Use school-level benchmarking as a strategy for data use. School-level benchmarking 
puts context around data use, making goals less abstract and more concrete. In the SREB 
programs, faculty are surveyed to determine if there is a common vision and mission. 
When the survey reports that 60 percent of the school faculty believe they can reach “all 
kids with the same thing a few kids got before,” it is clear the school is making a shift. 

Incorporate opportunity-to-learn in performance targeting . Participating schools and 
districts use the survey resources that are available from NAEP and in other widely 
distributed instruments to keep informed about how students view their learning 
environment and how teachers are changing their practices. They also compare student 
experiences at low- and high-performing schools, and let teachers know where the 
discrepancies are. They also compare what students say they are asked to do and what 
teachers say they are expecting of students. 

Clarify how middle schools are preparing students to succeed in high school. Middle 
schools should address what they are doing for students to succeed in a demanding 
curriculum in the high school. “If students tell you they are required to redo work until it 
meets the standard, that ... is an important indicator.” 

Identify students who may not be ready for high school and provide intervention . 
Focused assessment enables educators to identify fifth- or sixth-grade students who will 
need added assistance to prepare for high school. Anticipating these needs must start 
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well before high school, not in ninth grade. In addition, educators need to check to see 
that students are prepared to succeed in the basic high school math sequence of Algebra I 
and Geometry. 

Lessons from Experience: Emergent Themes 

The seminar concluded with suggestions of themes from the day’s discussion that have 
implications for the future use of educational performance targets. 

Conditions that limit the usefulness of performance targeting . Ms. Reisner observed that the 
tension between teacher ownership of targets and targets as sources of external accountability impedes the 
target setting process. “I didn’t find a simple solution — how you weigh the two sides of that tension — but 
that tension certainly affects the process for setting targets.” George Perry noted that “we still don’t have 
very good indicators about the real stuff that happens in schools,” ways to measure teacher and student 
belief systems and expectations, instructional content and behaviors, and classroom environments. With 
respect to content, schools should examine initial competencies and mastery in determining what the 
content of targets should be. Ms. Reisner reminded participants of the importance of using opportunity- 
to-leam indicators, as Mr. Bottoms had described, to ensure high school readiness. Measurement should 
be longitudinal and linked to high school. Districts should also be mindful of the incentives that may be 
set, whether intentionally or not, through performance targeting. 

Constructive use of performance targeting and available data. Targets can be informative at the 
school, classroom, and school levels, but data must be available for setting and reviewing progress 
benchmarks and for linking student performance to staff development. School-level benchmarking is a 
strategy for making targets more concrete for teachers and for putting them in a context for data use. 
However, target setting and use require focus. Mr. Perry called for districts to (1) establish what they 
need to gather information about and (2) communicate to others why the information is important. 
Districts need to justify the data they are collecting and use it well if they expect people to benefit from it. 
To affect teaching and learning, data reports must arrive in the classroom in a timely fashion and be 
presented in ways that are relevant to teachers. 

Linking state and local performance targets . Linking state and local performance targets is 
necessary to decrease duplicative data collection and reporting. “We can’t keep asking people to do 
different things. We have to figure out what’s important and what we need to learn,” Mr. Perry said. Ms. 
Reisner acknowledged that aligning the grading systems with test scores on high-stakes state assessments 
poses serious challenges to educators. These alignments would strengthen the linkages with policy levers 
such as high school readiness and, later, high school exit examinations. 
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New possibilities through technology. Mr. Feuer suggested that technology, which had not been 
very much part of the day’s conversation, holds real promise for providing assessment resources more 
efficiently and at less cost. Although technological solutions to the problems discussed during the day 
have not yet been found, scientists are beginning to use technology to devise richer assessments that 
reflect advances in cognitive science. There have been real advances in measurement theory and changes 
in the knowledge of how humans process information and make decisions. High speed technology “may 
provide ways to get at more parsimonious and economical methods of assessing” what we really want to 
know about learning, without compromising the quality of that information. “We’re not there yet,” Mr. 
Feuer said, and it will require investment, but he is optimistic about the possibility that “there might be 
some solutions.” 

Setting targets to generate shared commitment to accelerating achievement in middle-grades 
schools. Schools need to foster an “assessment culture,” in which thinking about assessments is 
embedded in educators’ jobs, as is linking student performance and staff development. Educators need to 
take responsibility for answering hard questions about persistent student needs and providing evidence 
through data that promote more meaningful accountability. Ms. Reisner called attention to the way that 
the HSTW program uses school-level benchmarks to put data use in context, making the information 
concrete so that practitioners can use it to inform their work. At the same time, Mr. Perry recognized that 
the responsible use of performance targeting also takes into account the limits of the data for teaching and 
learning. “At best, . . . student performance data can help set a direction,” but often there are other ways 
of collecting immediate, practical data along the way that would be “better indicators of how well 
teaching and learning” are occurring. 

Persevering to use performance targets in complex policy environments . Mr. Mizell reflected 
on the Foundation’s original hopes that target setting would provide focus for middle-school reform 
efforts. It took much more work for people to understand performance targets and standards than was 
originally realized. On the other hand, one of the outcomes of standards-based reform is getting the 
conversation started — asking hard questions, such as “are student performance goals necessary and 
useful?” Mr. Mizell challenged the districts to continue their efforts to engage the public and 
communicate “where it is necessary to go and why.” 

Improving communications. One area that none of the districts has addressed as much as they 
would have liked is public communication and engagement. The dialogue is still conducted at the level of 
public relations, rather than fully integrating the public into the reform effort. Ms. Winters pointed to the 
dilemma for districts: “For the public, the standard is performance on the test but to teachers it’s a 
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catechism of content,” and they are still struggling to understand that content component. This internal 
uncertainty limits teachers’ capacity to achieve the desired level of engagement. 

Keeping the conversation going. Mr. Mizell concluded the discussion by reminding the group 
that one of the primary benefits of the standards movement has been 4t to provoke conversation and deeper 
thinking than went on before and would go on otherwise.” Although the use and value of performance 
targeting in the Program for Student Achievement fell short of original expectations, it is clear that 
progress has been made. In many places, it is now possible for teachers and parents to receive concrete, 
standards-based information about students’ school performance. As a result, both groups know better 
what they can reasonably expect students to know and be able to do as they make transitions through 
school. 



Seminar participants will continue their discussion of the issues related to mechanisms for school 
improvement and change at the Foundation’s annual conference for sites participating in the Program for 
Student Achievement, which will occur in Pomona, California, October 28-30, 2001. 
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Issues in Setting and Using District Performance Targets 
An Edna McConnell Clark Foundation-Sponsored Seminar 

AGENDA 



June 25, 2001 
Washington, DC 



Meeting Goals: 



1. Promote an understanding of the issues in using district-level educational performance targets as 
incentives to improving teaching and learning 

2. Identify alternative approaches to performance targeting, which might be appropriate for use by 
districts and other sponsors of educational reform 



9:30 Introductions and Background for the Meeting Hayes Mizell 

Review of (1) the Foundation’s purpose in asking grantee districts to establish performance 
targets and (2) its assistance to districts in setting and using those targets 



9:45 Districts’ Experiences with the Clark Targets Corpus Christi 

Long Beach 
San Diego 

Each district will have 20 minutes to discuss its performance targets, its progress in achieving 

those targets, and the issues it encountered along the way, by addressing these questions: 

• What did the district learn from establishing performance targets and monitoring progress 
toward the targets over the past five years? 

• What were the conditions that enhanced or limited the effectiveness of the performance 
targets that the district set? How could these conditions have been usefully shaped in 
advance? 

• To what extent should local targets have been linked to state-established targets for 
student or school achievement levels? 



11:00 Break 

1 1: 15 Views from Consultants Suzanne Triplett, NCES 
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Michael Feuer, NAS 
Lloyd Bond, UNC-Greensboro 
Gene Bottoms, SREB 

Each consultant will have 15 minutes to comment on a district’s experiences and to suggest 
effective alternative mechanisms for promoting district- and school-level accountability. 

Panelists will address these issues: 

• Strengths and weaknesses of districts’ approaches 

• Alternative strategies for measuring student growth and change in educators’ practices 
that promote or reflect successful implementation of educational reform 

• Examples of indicators that work to promote improvement in complex policy 
environments 



12:30 Lunch 



1:15 Performance Targets: Challenges and Opportunities Gene Bottoms 

1:45 Districts’ Comments on Consultants’ Views 

Exchange of ideas among district participants and panelists on (1) the opportunities and dilemmas 
that shape performance targeting and (2) consultants’ proposed strategies and example indicators 

2:30 Principles for the Future Use of Educational Performance Targets Ellen Pechman 

Hayes Mizell 

Seminar participants will determine principles for the future use of educational performance 
targets, addressing the following issues: 

• Processes for setting targets 

• Content of targets 

• Measurement of progress toward targets 

• Use of targets to stimulate meaningful improvement in teaching and learning 
3:00 Next Steps 



3:30 Adjournment 
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